Skip to content

Make project identification the core functionality of this repository - #5

Merged
mbauman merged 26 commits into
JuliaPackaging:mainfrom
JuliaComputing:main
Nov 18, 2026
Merged

mbauman merged 26 commits into
JuliaPackaging:mainfrom
JuliaComputing:main

Conversation

@mbauman

@mbauman mbauman commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Adds identify() with provenance, project-aware vercmp via libversion_jll, a 3,031-entry regression corpus that checks project and version, and a scheduled Repology import that fits a standard GitHub runner and is gated on that corpus.


Why

SecurityAdvisories depends on mapping JLL sources to upstream projects and versions, and that mapping has been the fragile step. Expat 2.8.3 was recorded as 8_3 because a stale Repology pin missed it and the generic tag parser fell back to a guess. Nothing marked results as guesses, nothing caught regressions when the data changed, and the data only refreshed when someone ran the workflow by hand.

This makes Repology.jl the single place that answers "what upstream project and version is this source?". Every answer now says how it was obtained.

What changed

identify with provenance (src/identify.jl)

  • identify(url; commit=nothing, use_overrides=true) returns Identification(project, version, provenance).

  • identify_upstream(url, hash) is still there as a wrapper, so GeneralMetadata works without changes.

  • Resolvers run in a fixed order, and provenance records which one answered:

    provenance source
    override hand-verified exact sources in additional_info.toml (skipped when use_overrides=false)
    url-pattern curated url_patterns in additional_info.toml
    repology-url exact download-URL match in the Repology dump
    tag-scheme version parsed from a forge tag URL
    git-tag repo + commit resolved to a tag with git ls-remote
    repository project known from the repository URL, no version
    heuristic regex prefix match; consumers should treat it as low confidence
  • Version validation: a version is returned only if it is plausible for that project. Malformed values like 8_3 and placeholders like git come back as nothing.

  • Tag parsing: the parser handles mixed separators (R_2_8_3, REL_16_9, V_9_2_P1, OCCT V7_9_2, RELEASE.2025-…). It no longer mistakes all-digit date tags for commit hashes and no longer cuts versions short after a numeric prefix.

Curation in one file (additional_info.toml)

  • Each project can now have url_patterns, repositories, tag_patterns, cpes, version_scheme/version_flags and sources.
  • Curated url_patterns and repositories now take precedence over Repology's own data. That's needed where Repology assigns an upstream tarball or repository to a downstream subpackage, e.g. FFmpeg 7.1.1 → ffmpeg-static-libs-for-sunshine, rsync → rsync3, tiff → libtiff-5.
  • One repository can carry several projects: tag_patterns decide which one a tag belongs to. For example, libsdl-org/SDL.git gives sdl2 for release-2.* and sdl3 for release-3.*.
  • About 40 projects are curated so far, including giflib, ncurses, util-linux, msgpack-c, ffmpeg, libvirt, bitwarden-cli, rsync, tiff, quarto, openpmix, qdrant, wayland, the x.org xcb/util-macros tarballs, ghostscript, cudss, SDL2/SDL3, OpenPolicyAgent and TVM-FFI.

vercmp (src/vercmp.jl)

  • vercmp(a, b; flags) calls libversion's version_compare4 directly through libversion_jll.
  • vercmp(project, a, b) uses that project's version scheme:
    • version_scheme is libversion or semver, e.g. ImageMagick's 7.1.2-1 is a patch release after 7.1.2.
    • version_flags passes libversion flags, e.g. P_IS_PATCH for OpenSSH's 9.2p1.
  • No guessing: when a version has an ambiguous form (X-N, XpN, a letter suffix) that the scheme doesn't settle, vercmp(project, a, b) returns nothing. A form counts as settled if the project is curated or has already shipped a release in that form.
  • SemVer comparisons ignore build metadata.

Regression corpus (test/corpus.toml, test/known_failures.toml)

  • The corpus has 3,031 source → upstream mappings from GeneralMetadata (built by scripts/build_corpus.jl): 1,806 archives and 1,225 git+commit sources.

    • manual entries were hand-corrected in GeneralMetadata.
    • jll-version entries are ones where the upstream version equalled the JLL version.
    • Every entry must match both project and version. Checking only the version let a downstream project slip through for FFmpeg.
  • Where Repology was right and GeneralMetadata wrong, the corpus was corrected. Examples: xorriso 1.5.6.pl02 (not 1.5.6), cudss 0.7.0.20, deno (not deno-bin), qdrant (not rust:qdrant).

  • test/known_failures.toml lists the 6 current misses as @test_broken:

    • 5 are git URLs with no commit in GeneralMetadata.
    • SZ3 isn't in Repology.

    Regenerate it with scripts/known_failures.jl. Git entries are only checked when REPOLOGY_TEST_NETWORK is set.

    Results on the same data, without overrides:

    before: right / none / wrong after: right / none / wrong
    1,806 archives 1,499 / 307 / 0 1,748 / 58 / 0
    1,225 git+commit 1,174 / 28 / 23 1,195 / 30 / 0

    These numbers were measured before the later curation and the data refresh, which brought known misses down to 6.

Import fits a standard runner (scripts/update_repology_info.jl)

  • The script streams the zstd-compressed dump and parses only the 3 tables it uses, so there's no PostgreSQL restore, no libversion build and no disk-space workaround. The CSV dependency is gone too.
  • On the full dump it takes about 2–3 minutes, peaks at about 4 GB RAM and writes nothing to disk.
  • Cumulative exports: each dump only covers what distros ship today, so older release URLs would drop out of it. Entries from earlier exports are kept unless the new dump maps that URL differently.

Scheduled, corpus-gated refresh (.github/workflows/repology.yml)

  • Runs every Monday at 06:00 UTC and can be started by hand.
  • After the import it runs the tests against the new data with REPOLOGY_ALLOW_FIXED=1, then regenerates known_failures.toml.
    • Corpus passes (including known failures that are now fixed or give a different wrong answer): the data commit is pushed straight to the branch.
    • Any new regression, or any already-exported URL that now maps to a different project: the commit is force-pushed to automatic/repology-data-refresh-<branch>, and a PR is opened or updated. It shows the known_failures.toml diff and a table of project changes. The reviewer then fixes additional_info.toml, corrects test/corpus.toml, or accepts the change.
    • Repology renames projects often, so expect most weekly refreshes to come as PRs.
  • I tested this on the JuliaComputing fork: the first run found the xorriso corpus error (GeneralMetadata had 1.5.6 for xorriso-1.5.6.pl02.tar.gz) and opened [automatic] Repology data refresh with corpus regressions JuliaComputing/Repology.jl#3. That PR fixed the corpus entry and three downstream-subpackage mappings, and it is merged here, along with the project checks from Check the project of every corpus entry; review refreshes that move exported URLs to another project JuliaComputing/Repology.jl#4.

Data

  • data/ comes from the 2026-11 dump, generated by the workflow.

Settings needed on this repo

  • Settings → Actions → General → Workflow permissions: Read and write.
  • Settings → Actions → General: Allow GitHub Actions to create and approve pull requests. The refresh-PR path needs this.

Not in this PR

  • GeneralMetadata will move to rebuilding its upstream table from identify and storing provenance, and SecurityAdvisories will use provenance and vercmp(project, …). Both are separate changes.

mbauman and others added 26 commits November 18, 2026 20:23
…orpus

identify(url; commit) returns an Identification(project, version, provenance) and only
returns versions that look like plausible version numbers for the project. Hand-verified
corrections live in overrides.toml. test/corpus.toml holds 3,031 known-good mappings harvested
from GeneralMetadata.jl (scripts/build_corpus.jl) and gates the weekly data refresh.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…d-separator tags, refresh gate, per-commit known failures

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
… push

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…utrank Repology's URL data

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ose a PR when a refresh regresses it

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
… version 1.5.6.pl02

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…data had 1.5.6)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
… downstream subpackages

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…exported URL to another project through a PR

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ck between projects sharing a repository (SDL2/SDL3)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
… qdrant and cudss corpus entries

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…, letter-prefixed and _rc versions, and curated repositories/tag schemes for projects Repology's data dropped

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…SPIRV-LLVM-Translator and Win32-OpenSSH

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…e corpus passed and no exported URL changed project, otherwise propose a PR

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@mbauman
mbauman merged commit 3315c5f into JuliaPackaging:main Nov 18, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant